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namely, "that all state-supported institutions of higher education 
establish procedures and programs to measure student achievement*" By 
all accounts the English department of Virginia Military Institute 
(VMI) assumed an early and preemptive role in the assessment 
initiative* The assessment committee had devised a 3-part assessment 
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felt the scoring process, even for just 20 majors, would be unwieldy* 
Many worried that it would infringe on academic freedom by requiring 
a certain kind of writing assignment in department courses* Others 
were concerned about using a writing assessment instrument for work 
in a department concerned mainly with literature* In the end, 
portfolio assessment was voted on and passed* A committee has 
continued to work on remaining concerns* It has refined the process 
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The Writing Portfolio and English Program Assessment: 
Of Bumps, Bruises, and Lessons Learned 



In January 1986, the State Council of Higher Education for 
Virginia (SCHEV) presented our General Assembly with a 
commissioned report entitled "The Measurement of Student 
Achievement and the Assurance of Quality in Virginia Higher 
Education" (Senate Document No. 14). While acknowledging the 
difficulty and complexity of assessment as an endeavor, SCHEV 
felt that perhaps not enough was being done to address the 
public's "right to hold institutions of higher education 
accountable for effective teaching and learning" (15) . Opposed 
to "a system-wide minimum competency testing program" (15) , the 
Council nonetheless issued several "recommendations" with far- 
reaching implications for Virginia's colleges and universities: 
namely, "That all state-supported institutions of higher 
education establish procedures and programs to measure student 
achievement" (16), and that they "submit annual reports of 
progress in developing their assessment programs and concrete, 
non-anecdotal and quantifiable information on student 
achievement" (17). A colleague's handwritten notes from a 
meeting with SCHEV' s director suggest the consequences of 
enacting or ignoring these recommendations: "best hope for 

future- -Assessment . . . . our [higher education's] privileged 

position in state budget is history. . . . those who can make 

best case for funding will reap benefits." 

By the time I arrived at the Virginia Military Institute in 
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the fall. of 1992, the school was already well-heeled in observing 
what we have come to think of as the SCHEV Mandate. An Office of 
Assessment had been established almost immediately following the 
legislature's endorsement of SCHEV' s recommendations and, by 
1988, an interdisciplinary Assessment Committee had generated 
VMI's first full assessment report, an 83 -page document that 
said, in effect, "We're working on it." The Assessment Committee 
was actively constructing "a comprehensive program of formative 
evaluation" for the entire VMI experience (1, emphasis added), 
and at the department level, faculty committees were scrambling 
to discover and experiment with instruments that would assess the 
effectiveness of their programs both fairly and in terms that 
would satisfy SCHEV. The report indicated that "VMI has set a 
requirement that all departments be fully engaged in some form of 
assessment by the spring of 1990" (23) . 

By all accounts, my department, the Department of English 
and Fine Arts, assumed an early and proactive role in the 
assessment initiative. If I had to speculate, I would say this 
was largely due to our faculty' s strong belief in the preemptive 
effort- -a group disposition developed over years of weathering an 
intensely top-down administrative style in virtually all matters 
of importance. During the 1988-89 assessment effort, the 
department was one of four at VMI to participate in a pilot 
administration of the Educational Testing Service's new Major 
Field Achievement Test (MFAT) . Although the students "did not 
score significantly different from the national mean score," the 
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faculty were dissatisfied with "the outcomes" of the test (17) , 
not to mention wary of using such a purely "objective" instrument 
for evaluating a degree program in the liberal arts, and so they 
set out to create an instrument more reflective of the 
department's pedagogies and curricular goals. ^ Within two years, 
The English and Fine Arts Assessment Committee had devised a 
three-part assessment mechanism that it claimed would satisfy 
SCHEV: an alumni survey every five years; annual exit interviews 

with graduating seniors; and a senior capstone portfolio project. 
No part of this program should have been controversial; only the 
last part was,, and to some degree it remains so. 

The possibilities of the portfolio as an assessment tool 
were introduced in departmental meetings late in the 1989-1990 
academic year. Several of my colleagues had used portfolios in 
their composition classes, and they suggested that the concept 
could be adapted to yield the kind of "non-anecdotal and 
quantifiable" assessment information SCHEV was asking for. A 
capstone portfolio system would certainly serve as a more 
comprehensive and informative, not to mention a more humane, 
method of discovering our majors' various competencies than would 
any objective content-based test, like the MFAT. Their argument 
resembled Pat Belanoff's, who has said astutely that portfolios 
are not everything; they do not tell us everything; but still, 
"What they are is enough: a way of integrating testing, 

teaching, and curriculum" (20) . 

Resistance to the portfolio was minimal at first; after all. 



O 

ERIC 



5 



4 



it was preferable to the MFAT or the GRE Subject Test in Language 
and Literature, which loomed as untried possibilities.^ But very- 
soon, objections began to surface and became, indeed, the source 
of much internal strife. Some members of the department felt 
that the large-scale process of coordinating and team-scoring 
portfolios, even for just the twenty or so majors our department 
graduates each year, would be an unwieldy one. Others worried 
about how the portfolio might homogenize (as opposed to encourage 
a healthy consistency among) assignments and standards, and that 
it would ultimately infringe upon academic freedom by affecting 
the kinds of assignments we could make in our classes (e.g., more 
"creative" papers, hypertext, and the like) . And there was a 
quite fundamental obj ection- -one that dominated at least two 
hour-long department meetings - -based on the argument that there 
was. an inherent flaw with using a portfolio of writing to assess 
the work of a fairly traditional literature-oriented English 
department, whose primary curricular goal was still "to provide a 
comprehensive program of study for undergraduate majors in 
English," with an unequivocal emphasis on "the study of 
literature and the other arts" (VMI, "Goals") . Some faculty 
complained bluntly that the portfolio placed an "undue emphasis 
on writing--the 'service' arm of our department ... as opposed 
to teaching literature to majors" (VMI, "Memorandum" 2-16-93). 

In the fall of 1990, the English department nevertheless 
voted to adopt the senior capstone portfolio as the primary 
component of its tri-part assessment program. And on November 
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11, 1992 - -during my first semester as a member of the department, 
as well as a new and decidedly wide-eyed member of the Assessment 
Committee- -we took an even bolder step. Addressing an 
institutional directive that each department devise a senior 
project to test the competency of its majors, the department 
voted to make the portfolio not just an assessment tool: 
effective with the class of 1997, it would be a graduation 
requirement (VMI, "Letter" 11-17-92).^ 

Developing and enacting our senior capstone portfolio 
program has been a remarkably contentious and difficult 
enterprise virtually from the start; feathers ruffled in the 
process have not been completely smoothed even now, four full 
trial runs later. The requirement has met with degrees of 
opposition each year, usually just prior to the date the 
portfolios are due in mid spring, and usually from the same 
cluster of faculty who continue to believe the system is flawed. 
The department heard (and rejected) a proposal for a topical 
senior seminar, offered as a fairer and more sensible opportunity 
for seniors to demonstrate their competence in literary study as 
well as writing. We engaged in discussions about the lawsuits 
that could ensue by failing a portfolio of papers that had earned 
acceptable grades in the initial grading for our classes (and we 
in fact weathered exactly this problem- -vigorous complaints, but 
no lawsuit- -during our second trial year) . We debated faculty 
claims that the portfolio, with its score sheet and prescribed 
list of components, was in reality no better than any other 
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proficiency test, inevitably assessing form much more than 
substance. We even had a mass protest from one group of 
students, insisting that, since the portfolio was not vet a 
graduation requirement, we should not record the scores in their 
departmental records, and going on to question the ability of the 
faculty to render "an honest evaluation" of papers outside their 
"field[s] of expertise" (VMI , "Memorandum" 3-25-94). 

None of this has gone unobserved, and our Assessment 
Committee has worked diligently to address faculty as well as 
student concerns, all the while honing an assessment instrument 
that will do what it must : provide a clear window on the 

abilities and achievement-level of the majors who pass through 
our curriculum. We have, for instance, refined the process for 
submission as well as expanded the list of components for the 
capstone portfolio project. A portfolio must now include at 
least three complete papers representing at least two of four 
categories: a senior-year revision of an essay written for 

freshman composition (submitted with the original version for 
"value-added" comparison) ; a research essay written for an upper- 
level English or Fine Arts course; a non-research essay written 
for the same; or a personal essay on a topic created anew for 
each portfolio cycle. In addition, it must include a cover 
letter detailing the student's reasons for submitting each of the 
essays and reflecting on his (or her, depending on a forthcoming 
decision by the United States Supreme Court) development as a 
writer and an English major. To permit them the opportunity to 
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represent themselves and their abilities as fully as possible, we 
now invite students to include other kinds of work; however, to 
ensure some relative consistency among the material we evaluate, 
and thus to increase the validity of the instrument for whole- 
program assessment, we insist that this additional work cannot 
replace any of the required components. 

As for the process, beginning next year, students will 
submit their portfolios in October for anonymous evaluation by a 
randomly assigned team (usually two) of English and Fine Arts 
faculty. Anyone whose portfolio does not pass (i.e., receive an 
overall holistic score of 3 or better) will be allowed to revise 
and resubmit once, the following February; if the portfolio does 
not pass then. May graduation is impossible and a VMI degree in 
English can be awarded only the following year and only at the 
discretion of the department head. We urge close communication 
with the faculty advisor and serious efforts to revise papers for 
the portfolio in order to head off the dreaded extreme of this 
scenario. In addition, the Assessment Committee has taken an 
increasingly active role in the process, sponsoring two .or three 
informational sessions prior to the first date the portfolios are 
due in order to review the standards and procedure by which the 
students' work will be evaluated as well as our departmental 
goals for the assessment program. 

Because the stakes are so high for everyone involved, much 
of our attention has gone to the form that faculty teams use to 
derive a holistic score for each portfolio (See Appendix A) . It 
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is not yet perfect, but it does achieve something of a better 
balance among literary and rhetorical concerns than ever before, 
and it has the potential to yield the kind of reliable and 
relatively scientific data that should satisfy SCHEV' s need for 
such information. The form currently requires that portfolios be 
rated on a scale of 5 (excellent) to 1 (unacceptable) on six 
common points: Purpose ; Organization ; Development ; Mechanics : 

Context (defined as "an appropriate understanding of the social 
and cultural conditions and/or literary conventions informing the 
works in question"); and quality of the Cover Letter (VMI, 
"Criteria") . If a freshman paper is included, the quality of its 
revision is rated. There is also an unscored category called 
"Other, " which the score sheet identifies as a place for 
evaluators "to give greater importance to less tangible factors, 
such as evidence of intellectual interest for the subject in the 
papers" ("Criteria") and to comment on miscellaneous particulars 
that influenced the holistic score he or she awarded a portfolio. 
As an indication of how far we have come, this was a gesture to 
those who objected last year to the deletion of a formal category 
called "Enthusiasm, " which asked readers to divine how 
enthusiastically the papers had apparently been written and the 
portfolio compiled.^ 

Faculty, myself included, are still occasionally perplexed 




by the Context category; and we seem to be in agreement that not 
very much is learned in the rare instance that someone submits a. 
revised freshman paper. We will continue to refine it, but 
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overall the evaluation form is working well to indicate the 
relative quality of a portfolio as a representation of a 
student's ability to read, think, and write. If this past year's 
cycle is any indication, in fact, even some of my colleagues who 
resisted the portfolio have begun to take the process more 
seriously and participate more fully. The intensity of their 
opposition has faded noticeably. Perhaps they have accepted the 
fact that programmatic assessment is inevitable and acknowledged 
our department's citation as a "leading example" of assessment 
innovation within "the strongest [assessment] program in the 
state" (VMI, "Memorandum" 8-19-93; SCHEV, "Memorandum" 10-6-93). 

I would like to hope, though, that they have come around for 
another reason, that four years of practice with the program have 
served to assuage the single factor that many of us sense was at 
the heart of the resistance to the portfolio: an unpronounced 

but quite palpable fear- -of having colleagues inspect the kinds 
of work being done in our courses, of having an assessment tool 
shape the kinds of assignments we feel we can make, of being held 
accountable for quant if vincr the product being generated by our 
curriculum. It is a reasonable concern, certainly. But the 
Assessment Committee has addressed it with what has essentially 
been a public relations campaign, one that is being won a little 
more each year as faculty teams are surprised- -and relieved, I 
think- -to discover more similarity than difference in standards 
for determining excellence or deficiency and communicating those 
standards within our portfolio evaluation process. 
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We all acknowledge the fact that the data we have obtained 
so far are unreliable; the students have not taken the 
requirement very seriously, likely because they know it will not 
affect their graduation or even be recorded on their permanent 
records. They have also been too aware, some would say, of the 
shakiness behind the scenes as the requirement has been refined. 
We do have hope for the future, however: two of my four junior 

advisees have already talked to me about what will be required 
when they compile their portfolios next fall, for example, and 
their attitudes have not been grudging. I like to think they 
exemplify Ed White's assertion that students can be encouraged to 
view the portfolio as "inherently meaningful and worthwhile- -a 
record of work they have done and want to keep" (120) . That is 
what we tell them, at least; and we mean it. 

In the process, we hope that our portfolio requirement will 
develop as a "meaningful and worthwhile" enterprise for our 
department as well, providing information that will help us to' 
rethink our pedagogies and our curriculum. This year, the tenth 
anniversary of the SCHE'V Mandate, we are finally poised to reap 
the benefits of an assessment tool that we believe has developed 
as a most responsive and humane instrument for indicating how 
well our department does what it says it does. 
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Notes 

1. The 1988-89 assessment report allowed such innovation, 
claiming that "it must be held an inviolable rule that the 
nature, characteristics, and needs of the individual departments 
are the driving force of departmental assessment initiatives, not 
the reverse" (22) . 



2. There was reason to be nervous. SCHEV's recommendations were 
prefaced by the warning "that assessment can be costly. As 
institutions establish their programs, they will have to consider 
ways to minimize costs, by using information already available, 
by employing sampling techniques, and by adopting standardized 
tests of achievement where feasible " (15, emphasis added) . 



3. The original vote made the portfolio a requirement for the 
class of 1996, or the then-current class of freshmen ("Rats") . A 
later vote changed this to provide for four years of trial runs 
before implementing the portfolio as what Ed White calls a 
"barrier test" (130) . 



4 . It was this category that led people to cite (and even be 
forced to praise, I suppose) sentences like the one I stumbled on 
in a portfolio during this past evaluation cycle. In a non- 
research fine arts essay he included, a student wrote that a coda 
in Beethoven "commences softly and slow and low and is then built 
up as if grief has suddenly welled up from the bowels of the 
human soul and is ready to burst" (Portfolio #10) . 
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Department of English and Fine Arts 
Portfolio Assessment Grading Criteria 

Please mark the appropriate blocks when grading the portfolios. 



5 = excellent Name of Student: 

4 = above average 

3 = satisfactory Overall score (holistic) 

2 = below average 
1 = unacceptable 

An overall score of 3 is required for a passing grade. 

1 2 3 4 5 

I 1 1 1 1 1 Purpose: Papers should have an overall purpose 

(for example, demonstrate an interesting insight 

' ' ■ ' * ' or creative application, or address a real 

critical problem) . 

1 1 1 ] 1 Organization: Papers should have a logical 

organization that captures the reader's interest 
' ' ’ * ' ' and provides a sense of purpose and direction. 

Development: Papers should develop, illustrate 

1 ] 1 1 and explain all points: primary material should 

be discussed and analyzed rather than just 

summarized, and secondary material should. 

integrate into the organization and further the 
2L 3T O’ mn. s n t 



Mechanics: Papers should avoid clumsy, repetitive 
] or pretentious wording, and be free of errors in 
I spelling, grammar, punctuation and diction. If 
a research paper is included, the format and 
documentat i on ■ must be appropriate, correct and 

2.CCU2T3.tS . 



] Context: At least one paper other than the Rat 
I should show an appropriate understanding of the 
^ social and cultural conditions and/or literary 
conventions informing the works in question. 




Rat Paper Improvement: Is there improvement in 
style and focus? Does content appear more 
mature? Are revisions more than cosmetic? 



Cover Letter: Is it well-written, as befits an 
English major? Does it refer intelligently and 
specifically to each of the three papers it 
introduces? Does it contain reflection on the 

writer’s ODPent SS i cV> Tnn-^r^y9 




other: Evaluator, may comment on reverse about any 
other area important to this portfolio. 

16 





EXPLANATION OF THE SCORE SHEET 

1. The overall score is "holistic," which means it is not an 
average of the numbers in the boxes. Those numbers should serve as 
guidelines for the relative success of the portfolio in those 
specific categories, but, on the assumption that a portfolio is 
more than the sum of its parts, the professor will give a final 
score that reflects the overall quality of the portfolio. 

2. The seven named categories on the score sheet are all considered 
important by the English Department, but they are not necessarily 
all equally important. Faculty members will use their individual 
judgment when they weigh the significance of each category for a 
given portfolio. 

3. In addition, professors may use the category "Other" to give 
greater importance to less tangible factors, such as evidence of 
intellectual interest for the subject in the papers, and may thus 
give a portfolio a higher or lower grade than the raw numbers in 
the other categories would indicate. 
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